System and method for recognizing handwritten stroke input

ABSTRACT

A system, method and computer program product for use in recognizing content associated with handwritten stroke input to a computing device is provided. The computing device is connected to an input interface. A user is able to provide input by applying pressure to or gesturing above the input interface using a finger or an instrument such as a stylus or pen. The computing device has an input management system for recognizing content defined by the input. The input management system is configured to detect input of a handwritten stroke with respect to the interactive key layout, characterize the detected handwritten stroke by a reference stroke by determining a sequence of reference points associated with a sequence of interactive keys of the interactive key layout, assign probability scores to candidate characters, and cause recognition of sequences of characters by applying a language model in accordance with the assigned probability scores.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to European Application No. 16 290214.2 filed on Nov. 4, 2016, the entire contents of which isincorporated by reference herein.

TECHNICAL FIELD

The present description relates generally to the field of computer inputrecognition systems and methods using computing device interfaces. Thepresent description relates more specifically to systems and methods formanagement of typing input using handwriting recognition technology.

BACKGROUND

Computing devices continue to become more ubiquitous to daily life. Theytake the form of computer desktops, laptop computers, tablet computers,hybrid computers (2-in-1s), e-book readers, mobile phones, smartphones,wearable computers (including smartwatches, smart glasses/headsets),global positioning system (GPS) units, enterprise digital assistants(EDAs), personal digital assistants (PDAs), game consoles, and the like.Further, computing devices are being incorporated into vehicles andequipment, such as cars, trucks, farm equipment, manufacturingequipment, building environment control (e.g., lighting, HVAC), and homeand commercial appliances.

Computing devices generally consist of at least one processing element,such as a central processing unit (CPU), some form of memory, and inputand output devices. The variety of computing devices and theirsubsequent uses necessitate a variety of interfaces and input devices.One such input device is a touch sensitive surface such as a touchscreen or touch pad wherein user input is received through contactbetween the user's finger or an instrument such as a pen or stylus andthe touch sensitive surface. Another input device is an input surfacethat senses gestures made by a user above the input surface. A furtherinput device is a position detection system which detects the relativeposition of either touch or non-touch interactions with a non-touchphysical or virtual surface. Any of these methods of input can be usedgenerally for input through interaction with a displayed or graphical(virtual) keyboard through typing or “stroke-like” typing.

Virtual or ‘soft’ keyboards are widely used now and many options andapplications beyond those that come standard with portable (andnon-portable) computing devices are available, particularly for complexlanguages, such as those having many characters beyond those easilyplaced on a single keyboard panel or layout, for non-text characters,such as numbers, symbols and messaging characters (e.g., emoticons or“emojis”), and for increased speed.

With respect to increased speed, there is a growing popularity ofkeyboards which accept stroke-like input, such as the SWYPE®, SWIFTKEY®and TOUCHPAL® keyboards. These keyboards allow a continuous stroke to beinput over the virtual keyboard for selection of the underlying keys. Assuch, unlike traditional keyboard input in which each key is struck orinteracted with by a user for entry of text or commands, stroked‘keying’ inherently strikes more keys than necessary to input a word,for example. This leads to ambiguity in the input requiringdisambiguation processing to be performed.

Several techniques have been developed to provide this disambiguation.These conventional techniques include taking characteristics in thestroke path, such as bend, speed, direction, into account to determinethe intended keys, as disclosed for example in Canadian Patent No.2353159 and U.S. Pat. Nos. 7,098,896, 7,250,938 and 7,750,891, comparingthe stroke path to expected paths through key strings, as disclosed forexample in U.S. Pat. Nos. 7,895,518, 8,712,755 and 9,182,831, usingfuzzy logic and statistical models with respect to the keyboard layout,as disclosed for example in US Patent Application Publication No.2014/0359515 and U.S. Pat. Nos. 7,250,938, 7,750,891 and 8,667,414, orusing linguistic techniques, such as lexica and language models, asdisclosed for example in U.S. Pat. Nos. 8,756,499, 8,782,549, 8,843,845,9,182,831 and 9,454,240.

Each of these conventional techniques provide reasonable success inaccurately disambiguating stroke-typed input, including the ability tocomplete words and predict next words or characters, such as grammaticalsymbols. Each of these techniques however require the production ofunique recognition systems or engines for stroke-typing.

SUMMARY

The examples of the present disclosure that are described herein belowprovide systems and methods for recognizing content associated withhandwritten stroke input to computing devices. Each computing device isconnected to an input interface and has a processor, a memory and atleast one non-transitory computer readable medium for recognizingcontent under control of the processor.

In some examples, a system is provided in which the at least onenon-transitory computer readable medium is configured to detect input,with respect to an interactive key layout displayed on an interfacesurface of a computing device, of a handwritten stroke with respect tothe interactive key layout, characterize the detected handwritten strokeby a reference stroke by determining a sequence of reference pointsassociated with a sequence of interactive keys of the interactive keylayout, assign one or more probability scores to one or more candidatecharacters associated with each key of the interactive key sequence, andcause recognition of one or more sequences of characters by applying alanguage model in accordance with the assigned probability scores.

Each reference point may be determined in relation to the position ofone or more points on the detected handwritten stroke to acharacteristic of one or more of the interactive keys. Each referencepoint may be a centroid of the one or more points.

The at least one non-transitory computer readable medium may beconfigured to determine one or more combined probability scores of atleast a subset of candidate characters of a combination of consecutivepoints of the sequence of reference points, and cause recognition of oneor more sequences of characters by applying a language model inaccordance with the determined combined probability scores.

In some examples, a method is provided including the steps of detectinginput, with respect to an interactive key layout displayed on aninterface surface of a computing device, of a handwritten stroke withrespect to the interactive key layout, characterizing the detectedhandwritten stroke by a reference stroke by determining a sequence ofreference points associated with a sequence of interactive keys of theinteractive key layout, assigning one or more probability scores to oneor more candidate characters associated with each key of the interactivekey sequence, and causing recognition of one or more sequences ofcharacters by applying a language model in accordance with the assignedprobability scores.

Each reference point may be determined in relation to the position ofone or more points on the detected handwritten stroke to acharacteristic of one or more of the interactive keys. Each referencepoint may be a centroid of the one or more points.

The method may include determining one or more combined probabilityscores of at least a subset of candidate characters of a combination ofconsecutive points of the sequence of reference points, and causingrecognition of one or more sequences of characters by applying alanguage model in accordance with the determined combined probabilityscores.

In some examples, a non-transitory computer readable medium having acomputer readable program code embodied therein is provided. Thecomputer readable program code may be adapted to be executed toimplement a method including the steps of detecting input, with respectto an interactive key layout displayed on an interface surface of acomputing device, of a handwritten stroke with respect to theinteractive key layout, characterizing the detected handwritten strokeby a reference stroke by determining a sequence of reference pointsassociated with a sequence of interactive keys of the interactive keylayout, assigning one or more probability scores to one or morecandidate characters associated with each key of the interactive keysequence, and causing recognition of one or more sequences of charactersby applying a language model in accordance with the assigned probabilityscores.

Each reference point may be determined in relation to the position ofone or more points on the detected handwritten stroke to acharacteristic of one or more of the interactive keys. Each referencepoint may be a centroid of the one or more points.

The method may include determining one or more combined probabilityscores of at least a subset of candidate characters of a combination ofconsecutive points of the sequence of reference points, and causingrecognition of one or more sequences of characters by applying alanguage model in accordance with the determined combined probabilityscores.

BRIEF DESCRIPTION OF THE DRAWINGS

The present system and method will be more fully understood from thefollowing detailed description of the examples thereof, taken togetherwith the drawings. In the drawings like reference numerals depict likeelements. In the drawings:

FIG. 1 shows a block diagram of a computing device in accordance with anexample of the present system and method;

FIG. 2 shows a block diagram of example operating circuitry of thecomputing device;

FIG. 3 shows a schematic view of an input area provided by an inputmanagement system on a portion of an input interface of the computingdevice in accordance with an example of the present system and method;

FIG. 4 shows a schematic view of an example visual rendering of akeyboard layout in the input area for receiving typing input;

FIG. 5 shows the keyboard layout of FIG. 4 with stroke-typing inputdepicted thereon;

FIG. 6 shows a block diagram of a system for input recognition inaccordance with an example of the present system and method;

FIG. 7 shows a block diagram illustrating detail of the inputrecognition system of FIG. 6 in accordance with an example of thepresent system and method;

FIG. 8 shows the example stroke-typing input of FIG. 5 withcorresponding stroke reference points determined in accordance with anexample of the present system and method; and

FIG. 9 shows an example reference stroke including path portions linkingthe reference points of FIG. 8 determined in accordance with an exampleof the present system and method.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth by way of examples in order to provide a thorough understanding ofthe relevant teachings. However, it should be apparent to those skilledin the art that the present teachings may be practiced without suchdetails. In other instances, well known methods, procedures, components,and/or circuitry have been described at a relatively high-level, withoutdetail, in order to avoid unnecessarily obscuring aspects of the presentteachings.

The use of the term ‘text’ in the present description is understood asencompassing all alphanumeric characters, and strings thereof, in anywritten language and common place non-alphanumeric characters, e.g.,symbols, used in written text. Further, the term ‘non-text’ in thepresent description is understood as encompassing freeform handwrittenor hand-drawn content and rendered text and image data, as well asnon-alphanumeric characters, and strings thereof, and alphanumericcharacters, and strings thereof, which are used in non-text contexts.Furthermore, the examples shown in these drawings are in a left-to-rightwritten language context, and therefore any reference to positions canbe adapted for written languages having different directional formats.

The systems and methods described herein may utilize recognition ofusers' natural writing and drawing styles input to a computing devicevia an input interface, such as a touch sensitive screen, connected to,or of, the computing device or via an input device, such as a digitalpen or mouse, connected to the computing device or via a physical orvirtual surface monitored by a position detection system.

Whilst the various examples are described with respect to recognition ofhandwriting input using so-called online recognition techniques, it isunderstood that application is possible to other forms of input forrecognition, such as offline recognition in which images rather thandigital ink are recognized. The terms hand-drawing and handwriting areused interchangeably herein to define the creation of digital content byusers through use of their hands either directly onto a digital ordigitally connected medium or via an input tool, such as a hand-heldstylus. The term “hand” is used herein to provide concise description ofthe input techniques, however the use of other parts of a users' bodyfor similar input is included in this definition, such as foot, mouthand eye.

FIG. 1 shows a block diagram of an example computing or digital device100. The computing device may be a computer desktop, laptop computer,tablet computer, hybrid computers (2-in-1s), e-book reader, mobilephone, smartphone, wearable computer, digital watch, interactivewhiteboard, global positioning system (GPS) unit, enterprise digitalassistant (EDA), personal digital assistant (PDA), game console, or thelike. The computing device 100 includes components of at least oneprocessing element, some form of memory and input and/or output (I/O)devices. The components communicate with each other through inputs andoutputs, such as connectors, lines, buses, cables, buffers,electromagnetic links, networks, modems, transducers, IR ports,antennas, or others known to those of ordinary skill in the art.

The illustrated example of the computing device 100 has at least onedisplay 102 for outputting data from the computing device such asimages, text, and video. The display 102 may use LCD, plasma, LED,iOLED, CRT, or any other appropriate technology that is or is not touchsensitive as known to those of ordinary skill in the art. At least someof the display 102 is co-located with at least one input interface 104.The input interface 104 may be a surface employing technology such asresistive, surface acoustic wave, capacitive, infrared grid, infraredacrylic projection, optical imaging, dispersive signal technology,acoustic pulse recognition, or any other appropriate technology as knownto those of ordinary skill in the art to receive user input. The inputinterface 104 may be bounded by a permanent or video-generated borderthat clearly identifies its boundaries. Instead of, or additional to, anon-board display, the computing device 100 may have a projected displaycapability or is able to operate with a projected display, such that theinput interface is a virtual surface. Further, the display itself may beseparate from and connected to the computing device.

The computing device 100 may include one or more additional I/O devices(or peripherals) that are communicatively coupled via a local interface.The additional I/O devices may include input devices such as a keyboard,mouse, scanner, microphone, touchpads, bar code readers, laser readers,radio-frequency device readers, or any other appropriate technologyknown to those of ordinary skill in the art. Further, the I/O devicesmay include output devices such as a printer, bar code printers, or anyother appropriate technology known to those of ordinary skill in theart. Furthermore, the I/O devices may include communications devicesthat communicate both inputs and outputs such as a modulator/demodulator(modem; for accessing another device, system, or network), a radiofrequency (RF) or other transceiver, a telephonic interface, a bridge, arouter, or any other appropriate technology known to those of ordinaryskill in the art. The local interface may have additional elements toenable communications, such as controllers, buffers (caches), drivers,repeaters, and receivers, which are omitted for simplicity but known tothose of skill in the art. Further, the local interface may includeaddress, control, and/or data connections to enable appropriatecommunications among the other computer components.

The computing device 100 has operating circuitry 105. FIG. 2 shows ablock diagram of an example of the operating circuitry 105. In thisexample, the operating circuitry 105 includes a processor 106, which isa hardware device for executing software, particularly software storedin a memory 108. The processor can be any custom made or commerciallyavailable general purpose processor, a central processing unit (CPU),commercially available microprocessors including a semiconductor basedmicroprocessor (in the form of a microchip or chipset), microcontroller,digital signal processor (DSP), application specific integrated circuit(ASIC), field programmable gate array (FPGA) or other programmable logicdevice, discrete gate or transistor logic, discrete hardware components,state machine, or any combination thereof designed for executingsoftware instructions known to those of ordinary skill in the art.

The memory 108 can include any one or a combination of volatile memoryelements (e.g., random access memory (RAM, such as DRAM, SRAM, orSDRAM)) and nonvolatile memory elements (e.g., ROM, EPROM, flash PROM,EEPROM, hard drive, magnetic or optical tape, memory registers, CD-ROM,WORM, DVD, redundant array of inexpensive disks (RAID), another directaccess storage device (DASD), or any other magnetic, resistive orphase-change nonvolatile memory). Moreover, the memory 108 mayincorporate electronic, magnetic, optical, and/or other types of storagemedia. The memory 108 can have a distributed architecture where variouscomponents are situated remote from one another but can also be accessedby the processor 106. Further, the memory 108 may be remote from thedevice, such as at a server or cloud-based system, which is remotelyaccessible by the computing device 100. The memory 108 is coupled to theprocessor 106, so the processor 106 can read information from and writeinformation to the memory 108. In the alternative, the memory 108 may beintegral to the processor 106. In another example, the processor 106 andthe memory 108 may both reside in a single ASIC or other integratedcircuit.

The software in the memory 108 includes an operating system 110, aninput management system 112 and an input recognition system 114, whichmay each include one or more separate computer programs. Each of thesehas an ordered listing of executable instructions for implementinglogical functions. The operating system 110 controls the execution ofthe input management system 112 and the input recognition system 114, ormay incorporate the functions of these systems. The operating system 110may be any proprietary operating system or a commercially or freelyavailable operating system, such as WEBOS, WINDOWS®, MAC and IPHONE OS®,LINUX, and ANDROID. It is understood that other operating systems mayalso be utilized. Alternatively, the input management system 112 andinput recognition system 114 of the present system and method may beprovided without use of an operating system.

The input management system 112 includes one or more processing elementsrelated to detection, management and treatment of user input. Thesoftware may also include one or more applications related to inputrecognition, different functions, or both. Some examples of otherapplications include a text editor, telephone dialer, contactsdirectory, instant messaging facility, computer-aided design (CAD)program, email program, word processing program, web browser, andcamera. The input management system 112, and the applications, includeprogram(s) provided with the computing device 100 upon manufacture andmay further include programs uploaded or downloaded into the computingdevice 100 after manufacture.

The input management system 112 of the present system and method managesinput into the computing device 100 via the input interface 104, forexample. Input is managed through the provision of input tools to usersand the handling of the input for processing and the like. The inputtools include the provision and display of dedicated input areas on theinput interface 104 or the provision of the (substantially) entire inputinterface 104 for the receipt of user input via interaction with or inrelation to the input interface 104. The dimensions and functionality ofthese input areas are provided in correspondence with, and responsiveto, the dimensions and orientation of the display area of the devicedisplay 102 in a manner well understood by those skilled in the art.

FIG. 3 shows an example input area 300 displayed by the input managementsystem 112 on the input interface 104. The input area 300 is an area orzone of the display of the computing device 100 which is to be used forthe input of content and/or control gestures by users. In otherexamples, substantially all of the input interface may be configured asthe input area. Any digital device user is already accustomed togesturing on screen to type or write content and to enter controlcommands for application and device navigation, content editing, etc.Such interactions with the input interface 104 of the computing device100 are generally detected by the processor 106 and this information iscommunicated to the input management system 112 for interpretation andrecognition processing.

The input area 300 is configured to receive user input throughsingle-point or single-position gestures or interactions, such as tap,short and long press, multi-point or multi-position gestures orinteractions, such as double tap, and stroke gestures, such as swipe. Inorder to translate these gestures to input of specific content orcommands, an interactive (virtual) keyboard panel 400 may be displayedin the input area 300, such as the ‘azerty’ style keyboard layoutvariant of the ‘qwerty’ key or keyboard layout shown in FIG. 4.

The illustrated layout of the keyboard panel 400 is merely an example,and many other known keyboard layouts and methods, e.g., qwerty orazerty mapped layouts for language specific variants like BoPoMoFo,Hangul, JIS, Hanyu Pinyin, phonetic, non-qwerty layouts for differentlanguages like Jcuken, InScript, reduced keyboard, such as T9 or T12, oryet-to-be-developed keyboard layouts, are applicable to the presentsystem and method used either singularly with respect to the computingdevice or selectively (discussed in detail later) by storage ofdifferent keyboard layouts in the memory 108, for example. Further,layouts that provide access to non-alphabetic characters, such asnumerals, grammatical marks, emojis, etc. are also applicable, typicallyselectively.

As discussed, the example keyboard panel 400 includes an interactive keyor keyboard layout. The keyboard layout has content keys 402 which wheninteracted with by users, such as through a single-point gesture or‘strike’ thereon or over, result in the input of content, and commandkeys 404 which when interacted with by users, such as through asingle-point gesture or strike thereon or over, result in the input ofcontrol commands, e.g., applying a tap on the “backspace” key causes thebackspacing deletion of previously input characters, or launching ofkeyboard sub- or dedicated layouts, e.g., special character layoutshaving keys for numerals, grammatical marks, emojis, language specificlayouts as described above, language alternatives layouts providingaccess to accents, character alternatives based on strokes, etc. Boththe content and command keys are generally displayed with characterdepictions corresponding to the content or command input which resultsfrom interaction with that key.

Users may provide input with respect to the keyboard panel using afinger or some instrument such as a pen or stylus suitable for use withthe input interface, and this input is detected by the input managementsystem 112. The user may also provide input by making a gesture abovethe input interface 104 if technology that senses or images motion inthe vicinity of the input interface 104 is being used, or with aperipheral device of the computing device 100, such as a mouse orjoystick, or with a projected interface, e.g., image processing of apassive plane surface to determine the input sequence and gesturesignals.

The present system and method handles the user keyboard input to providean input signal of a typing stroke which is determined by the presentsystem and method as a sequence of points characterized by at least thestroke initiation location, the stroke termination location, and thepath connecting the stroke initiation and termination locations ascaptured by the input management system 112 and/or input recognitionsystem 114. Further information such as timing, pressure, angle at anumber of sample points along the path may also be captured to providedeeper detail of the keyboard strokes. The input management system 112in conjunction with the input recognition system 114 recognizes one ormore sequences of keys of the keyboard layout corresponding to thedetermined sequences of points.

FIG. 5 shows the keyboard panel 400 with example stroke-typing input500. The stroke-typing input 500 includes a single continuous strokewhich successively passes over or on (or near) the keys “e”, “d”, (“f”),“c”, “v”, (“g”), “f”, “d”, “e”, “r”, “t”, “y”, “t”, “g”, “h”, “j”, “i”,“k”, (“j”), “n”, “b”, “v” and “g”. Clearly these keys on the path of thestroke do not all together form a word or other textual input, and assuch the input management system 112 determines which of these keys (orother keys) the user intended to interact with in order to inputcontent, such as one or more text elements. The manner of thisdetermination is now described.

The input management system 112 may be configured to (temporarily)render the stroke-typing input 500 as so-called “digital ink” on thekeyboard panel 400, similar to the depiction in FIG. 5, for example.This display of the digital ink provides feedback to users that thestroke-typed path has been received by the input management system 112.The input of content via the input area 300 may cause the rendereddisplay of the content elsewhere on the display 102, such as a componentof an active application of the computing device 100, in ‘typeset ink’(e.g., fontified text), for example. In the present system and methodthe input management system 112 causes display of the input contenteither directly or via communication of the input to the activeapplication and/or operating system 110, for example, in a mannersimilar to that conventionally employed by operating systems andcomponents and applications thereof. That is, for typing input thecontent is rendered as digital objects, e.g., typeset ink. The displayedcontent is content which has been recognized and interpreted by theinput recognition system 114. As such, the present system and method mayproduce the interpreted content as ink objects.

Ink objects include links between the rendered display of the typesetink and the recognition candidates produced by the recognitionprocessing, so that the displayed content is provided as interactiveink. This may be achieved as described in U.S. patent application Ser.No. 15/083,195 titled “System and Method for Digital Ink Interactivity”filed claiming a priority date of 7 Jan. 2016 in the name of the presentApplicant and Assignee, the entire contents of which is incorporated byreference herein.

To achieve display of content, the input management system 112 isconfigured to detect the input of typing at the input area 300 and causethe input content (or commands) to be recognized by the inputrecognition system 114 under control of the processor 106, for example.The input recognition system 114 and any of its components, with supportand compliance capabilities, may be a source program, executable program(object code), script, application, or any other entity having a set ofinstructions to be performed. When a source program, the program needsto be translated via a compiler, assembler, interpreter, or the like,which may or may not be included within the memory 108, so as to operateproperly in connection with the operating system 110.

Furthermore, the input recognition system with support and compliancecapabilities can be written as (a) an object oriented programminglanguage, which has classes of data and methods; (b) a procedureprogramming language, which has routines, subroutines, and/or functions,for example but not limited to C, C++, Pascal, Basic, Fortran, Cobol,Perl, Java, Objective C, Swift, Python, C# and Ada; or (c) functionalprograming languages for example but not limited to Hope, Rex, CommonLisp, Scheme, Clojure, Racket, Erlang, OCaml, Haskell, Prolog, and F#.

Alternatively, the input recognition system 114 may be a method orsystem for communication with an input recognition system remote fromthe device, such as server or cloud-based system, but is remotelyaccessible by the computing device 100 through communications linksusing the afore-mentioned communications I/O devices of the computingdevice 100. Further, the input management system 112 and the inputrecognition system 114 may operate together or be combined as a singlesystem.

With respect to typing input, sequences of points entered on or via theinput interface 104 are processed by the processor 106 and routed to theinput recognition system 114 for recognition processing, and withrespect to stroke-typing input, the continuous stroke entered on or viathe input interface 104 are processed by the processor 106 and routed tothe input recognition system 114 for recognition processing. Becausedifferent users may naturally type the same text with slight variations,the input recognition system 114 accommodates a variety of ways in whicheach object may be entered whilst being detected as the correct orintended object.

The present Applicant and Assignee has researched and developedhandwriting recognition technology over a substantial number of years,and in so doing honed the accuracy and reliability of the underlyinghandwriting recognition engine. This handwriting recognition is based inpart on analysis of the handwritten strokes making up the content. Inthe present system and method the recognition processing performed onhandwritten input of content, e.g., text, is adapted to processingstroke-typing input, where the ‘strokes’ of the typing input are treatedsimilarly to the ‘strokes’ of handwriting input. In this way a ‘single’multi-modal recognition system that is able to recognition process bothhandwriting and stroke-typing input is provided, thus obviatingprovision of a specific recognition engine for either input. This is nowdescribed.

FIG. 6 is a schematic pictorial of an example of the input recognitionsystem 114. The input recognition system 114 includes stages such aspreprocessing 116, candidate selection 118 and output 120. These stagesare part of the handwriting recognition system and are adapted toprocess the detected typing input in relation to the keyboard layoutbeing used.

The preprocessing stage 116 processes the typing input signal (typedink) to achieve greater accuracy and to reduce processing time duringthe candidate selection stage 118. This preprocessing may includere-sampling/normalizing (using a background layout), smoothing, andclustering of points. The preprocessed sequences are then passed to thecandidate selection stage 118. It is understood that the preprocessingstage may be provided to the input recognition system 114 by anothersource, such as an optical character recognizer. Further, it isunderstood that the preprocessing stage may not be employed by the inputrecognition system 114 if the keyboard input signal is capable of beingrecognition processed without such preprocessing.

The candidate selection stage 118 may include different processingelements or experts. FIG. 7 is a schematic pictorial of the example ofFIG. 6 showing schematic detail of the candidate selection stage 118.Three experts, a segmentation expert 122, a character expert 124, and alanguage expert 126, are illustrated which collaborate through dynamicprogramming to generate the output 120.

The segmentation expert 122 defines the different ways to segment theinput signals into individual element hypotheses which form sequences ofelements as a segmentation graph in accordance with interactive key orkeyboard layout information 128, which may be stored by the memory 108of the computing or digital device 100, for example. For single-pointinteractions with the keyboard panel 400 (e.g., key typing) the elementhypotheses are formed in sequences of mandatory points, whereas formulti-point interactions with the keyboard panel 400 (e.g.,stroke-typing) the element hypotheses are formed as re-sampled sequencesof optional points (described in detail later).

The layout information 128 is provided to, or determined by, the inputrecognition system 114 from a plurality of possible keyboard layoutsthat could be displayed as stored in the memory 108 of the digitaldevice 100, for example. Accordingly, the keyboard layout which is beinginteracted with in the input panel 400 is known, and therefore therelative positions of the detected points in the input panel 400 aremapped to the keys 402 and/or 404 as keystrokes.

Further, because a virtual keyboard does not have physical ‘keys’ and/orthe size of the device display 102 may limit the size of the keys withinthe keyboard layout and the spacing therebetween, it is possible thatusers will strike more than one key substantially simultaneously whenkey typing, strike the wrong key when key typing or stroke-over wrongkeys or miss keys when stroke-typing. Such multiple keystrokes makeinterpretation of the intended key ‘press’ or stroke-over uncertain.

Thus, from the mapping, the input recognition system 114 may alsodetermine the keys 402/404 which neighbor the detected points.Accordingly, in an example of the present system and method thesegmentation graph is produced with paths having nodes according toelement hypotheses produced for each or some of these neighboring keysas well (thereby implementing so-called ‘fuzzy’ logic, described in moredetail later).

The character expert 124 provides or assigns probability scores forcharacters according to the input signal and the layout information 128and outputs a list of element candidates with probabilities or scoresfor each node of the segmentation graph (described in detail later).

The language expert 126 generates linguistic meaning for the differentpaths in the segmentation graph using language models (e.g., grammar,semantics) of the linguistic resource. The language expert 126 checksthe candidates suggested by the other experts according to linguisticinformation provided by a language component 130. The linguisticinformation can include a lexicon, regular expressions, etc., and is thestorage for all static data used by the language expert 126 to execute alanguage model. Possible forms of this language model are for examplebut not limited to n-gram, artificial neural networks, recurrentartificial neural networks including long short-term memory networks.

The language expert 126 aims at finding the best recognition path. Inone example, the language expert 126 does this by exploring the languagemodel representing the content of linguistic information. In addition toa lexicon constraint, for example, the language model can rely onstatistical information, such as finite state automaton (FSA), on one ormore given languages which models for how frequent a given sequence ofelements appears in the specified language or is used by a specific userto evaluate the linguistic likelihood of the interpretation of a givenpath of the segmentation graph. The linguistic information issubstantially computed off-line, with or without adaption according tothe results of recognition and user interactions, and provided to thelanguage expert 126.

After recognition processing, the selected content is provided as theoutput 120 to the input management system 112. The input managementsystem 112 may then render the output 120 on the display 102 asdescribed earlier, including being included in a list of likely contentcandidates.

FIGS. 8-9 show various details of example processing applied to thestroke-typing input 500 of FIG. 5 by the input recognition system 114,particularly with respect to the processing performed by thesegmentation and character experts 122 and 124 of the candidateselection stage 118.

With respect to handwriting input recognition, the segmentation expert122 may form element hypotheses by grouping consecutive segments orsnippets of the original input handwritten strokes, so-called ‘raw ink’to obtain the segmentation graph where each node corresponds to at leastone element hypothesis and where adjacency constraints between elementsare handled by the node connections. In this segmentation, particularpoints in each stroke for defining the segments are pre-determined(e.g., based on pre-defined absolute or percentage distances betweenpoints) or dynamically determined (e.g., based on characteristics of thestrokes, such as shape, changes in direction, weight or thickness,height or length). This technique may result in ‘over-segmentation’ ofthe strokes, which may be optimized or reduced based on knownhandwritten character parameters, for example. In the present system andmethod, the segmentation expert 122 applies similar segmentation to thestroke of the stroke-typing input by re-sampling the typed stroke.

For example, FIG. 8 shows the stroke-typing input 500 as depicted inFIG. 5 but with the keyboard panel 400 de-emphasized for illustrationpurposes only. As described earlier, the input management system 112provides the single continuous stroke of the stroke-typing input 500 tothe input recognition system 114. The segmentation expert 122 analyzesthe path of the stroke 500 to determine a sequence of points 800 inrelation to the path for segmentation of the stroke 500. This sequenceof points 800 is determined for stroke-typing based on the keyboardlayout information 128 rather than parameters of handwritten charactersas in the case of handwriting recognition. That is, the segmentationexpert 122 determines at least one point in relation to the stroke path500 for each of the keys according to the keyboard layout information128 that the path passes directly over or substantially near, e.g.,within a pre-defined and/or settable (such as via the user interface(UI)) distance.

These points may be determined in a number of ways. For example, in someof the conventional techniques described earlier, points on the rawstroke or trace are used based on closest distance to dimensionalelements of each key, e.g., the center of the key, edges of the key,area of the key. However, such ‘re-sampling’ may cause inherent errorsin the stroke being included or emphasized, such as excessive ‘wobble’or ‘wiggle’, excessive deviations, directional errors, etc. Accordingly,the present system and method determines the sequence of points in amanner which reduces the influence of such errors. This is achieved bydetermining a common or reference point for a group or range of pointson the path, the position of which is compared or related to the keys ofthe keyboard layout. In this way, a reference stroke is determined whichcharacterizes the input handwritten stroke for recognition processing bythe handwriting recognition system.

In the illustrated example of FIG. 8 the reference points are determinedby the input recognition system 114 as the centroids of consecutivere-sampled points of the raw stroke that share the same closest key. Theconsecutive points may be continuous or have distances therebetween aspre-defined, dynamically defined, and/or settable, through the UI forexample, to provide optimal segmentation.

As can be seen from FIG. 8, the sequence of reference points 800 isdetermined to include a reference point 801 related to the key “e”, areference point 802 related to the key “d”, a reference point 803related to the key “f”, a reference point 804 related to the key “c”, areference point 805 related to the key “v”, a reference point 806related to the key “g”, a reference point 807 related to the key “f”, areference point 808 related to the key “d”, a reference point 809related to the key “e”, a reference point 810 related to the key “r”, areference point 811 related to the key “t”, a reference point 812related to the key “y”, a reference point 813 related to the key “t”, areference point 814 related to the key “g”, a reference point 815related to the key “h”, a reference point 816 related to the key “j”, areference point 817 related to the key “i”, a reference point 818related to the key “k”, a reference point 819 related to the key “j”, areference point 820 related to the key “n”, a reference point 821related to the key “b”, a reference point 822 related to the key “v” anda reference point 823 related to the key “g” in succession.

It is understood that parameters other than centroids for determiningthe common points are possible, such as the point at mean or averagedistance, from the shared closest key, the central point in the range ofpoints having a shared closest key. Also, ranges of points other thanthose having a shared closest key may be used, such as only points on aportion of the path that overlay a key.

Upon determining the reference points the input recognition system 114,or the input management system 112, may also construct the sequence ofreference points as a reference stroke by linking or connecting thedetermined points. In the simplest form, these reference stroke portionsor segments are defined as lineal connections between consecutivepoints.

For example, FIG. 9 shows the reference stroke 800 as determined toinclude a stroke portion 824 between the points 801 and 802, a strokeportion 825 between the points 802 and 803, a stroke portion 826 betweenthe points 803 and 804, a stroke portion 827 between the points 804 and805, a stroke portion 828 between the points 805 and 806, a strokeportion 829 between the points 806 and 807, a stroke portion 830 betweenthe points 807 and 808, a stroke portion 831 between the points 808 and809, a stroke portion 832 between the points 809 and 810, a strokeportion 833 between the points 810 and 811, a stroke portion 834 betweenthe points 811 and 812, a stroke portion 835 between the points 812 and813, a stroke portion 836 between the points 813 and 814, a strokeportion 837 between the points 814 and 815, a stroke portion 838 betweenthe points 815 and 816, a stroke portion 839 between the points 816 and817, a stroke portion 840 between the points 817 and 818, a strokeportion 841 between the points 818 and 819, a stroke portion 842 betweenthe points 819 and 820, a stroke portion 843 between the points 820 and821, a stroke portion 844 between the points 821 and 822 and a strokeportion 845 between the points 822 and 823.

In the drawings, the reference stroke portions or segments are shown indashed lines and the reference points are shown as points. It isunderstood that this is for illustration purposes only and the referencestroke is not necessarily displayed on the input interface 104 of thedevice 100.

As can be seen, the reference stroke 800 is a representation of theinput stroke 500, with stroke portions or segments defined bysegmentation points. In this way, the input stroke for typing keys 402of the keyboard panel 400 is segmented by the segmentation expert 122 ina similar manner to handwritten strokes of characters. With respect torecognition of such handwriting, the input recognition system 114formulates and tests character hypotheses for each of these segments andgroups of these segments.

In particular, the character expert 124 provides classification of thefeatures of the handwritten characters extracted by a characterclassifier (not shown) and outputs a list of element candidates withprobabilities or recognition scores for each node of the segmentationgraph determined by the segmentation expert 122. That is, the result foreach node of the segmentation graph, e.g., each segmentation point andassociated segment (or groups thereof), is a set of possible characterswith associated probabilities based on the characteristics of thatsegment (or group of segments).

The classifier may incorporate information related to characteristics ofhandwritten characters, such as shape, slant, etc. which assists thecandidate selection stage 118 in recognizing characters of candidatessuggested by the other experts 122 and 126. Many types of classifierscould be used to address this recognition task, e.g., Support VectorMachines, Hidden Markov Models, or Neural Networks such as MultilayerPerceptrons, Deep, Convolutional or Recurrent Neural Networks. Thechoice depends on the complexity, accuracy, and speed desired for thetask.

In the present system and method, the character expert 124 utilizes thekeyboard layout information 128 in a similar manner to the characterclassifier. That is, the layout information 128 provides the charactersor commands (such as, keyboard layout change, menu launching and editingoperations on the displayed recognized content, for example) assigned toeach of the keys 402/404 of the displayed keyboard panel 400. Thus, fromthe layout-to-point mapping for the reference points of the referencestroke determined by the segmentation expert 122, the input recognitionsystem 114 determines the character(s) or functions corresponding to thekeys 402/404 and outputs a list of element candidates with probabilitiesor recognition scores for each of the nodes of the segmentation graph.That is, the result for each node of the segmentation graph, e.g., eachreference point and associated reference stroke segment, may be a set ofpossible keys (characters) with associated probabilities based on thecharacteristics of reference stroke.

For example, for the reference point 801, the key candidate listdetermined by the character expert 124 from the keyboard layoutinformation 128 includes the key 402 corresponding to the character “e”and the adjacent keys 402 corresponding to the characters “z”, “r”, “s”and “d”. The associated probability scores for these candidates may havethe character “e” ranked highest, as the reference point 801 is on thecharacter “e” key 402, the character “d” ranked next, as the referencepoint 801 is closer to the character “d” key 402 than the character “z”,“s” and “r” keys 402 (in this determination it may also be taken in toaccount that the segment 824 associated with the reference point 801progresses over the character “d” key 402), and the other characters“z”, “s” and “r” ranked lowest. In this way, a type of fuzzy logic isapplied.

The character expert 124 may adjust the probability scores for thesefuzzy points and add character alternatives based on surrounding keys ofthe layout, keys that may or may not be included for stroke-typing,and/or for those not directly accessible through the displayed layout(e.g., accented variants of characters, like é, è, ê for e). This can bedone for all detected points, e.g., for all element hypothesesrepresenting all nodes of the segmentation graph, or for only thosepoints that are considered fuzzy, e.g., the detected point is far fromthe center of a key.

The character expert 124 may incorporate additional information of thestroke path, such as track starting and termination points and points ofinflection in the path spatially and/or temporally (e.g., changes indirection, curvature, slowing down and speeding up), in order to adjustthe probability score of each possible character, which assists thecandidate selection stage 118 in recognizing characters of candidatessuggested by the other experts 122 and 126.

Keyboard layout change may also be provided by interaction with theinput panel 400 such as input of a multiple-point gesture, like swiping,in order to ‘reveal’ display of different keyboard layouts. Further, akeyboard layout may provide access to alternatives of the displayedcharacter keys, such as accents, upper/lower case, language changes,symbols, numbers, etc., through multiple interactions or long-press orpressure interactions with single keys, particularly on reduced sizekeyboard layouts having limited keys displayed.

As discussed earlier, the segmentation applied for handwritingrecognition leads to over-segmentation, e.g., too many paths in thesegmentation graph. This enhances the accuracy of the handwritingrecognition through best segmentation path selection based on dynamicprocessing of the character and language experts 124 and 126 on thehypotheses dynamically provided by the segmentation expert 122. There-sampling performed by the present system and method to provide there-sampled or reference stroke similarly over-segments the input typingstroke in order to achieve similar enhanced accuracy.

In the handwriting recognition processing, consecutive segments aresuccessively grouped and the segmentation graph is built with multiplepaths having nodes defining the differently grouped segments. Thecandidate characters and associated probabilities of each of these nodesare provided by the character expert 124 through consideration of there-formed or combined segments by the character classifier, as describedearlier. In the present system and method the segmentation expert 122similarly successively groups consecutive segments of the referencestroke to build the segmentation graph with multiple node paths, and thecharacter expert 124, or the input recognition system 114, combines thecandidate keys and associated probability scores of the reference pointsof the combined segments in order to provide a combined list ofcandidate keys and associated probabilities. The scored segmentationgraph is analyzed by the language expert 126 as described earlier.

This combination may be performed in a number of ways depending on thelevel of precision required. For example, the combined candidates may beat least a subset of the available candidates, such as only thecandidates of the last reference point (in time-order) of the combinedsegments or the highest ranked candidates with combined probabilityscores.

The former case omits information of the reference points betweensubsequent nodes in the segmentation graph, thereby skipping thosepoints. For example, from the reference stroke 800 of FIG. 8, thesegmentation graph may include a path with the reference point 801 as aninitial (first) node and a (second) consecutive node having the keycandidates and probability scores determined for the reference point805, but not those determined for the intervening reference points 802,803 and 804.

The latter case adjusts the key candidate list based on re-calculationof the associated probabilities. For example, from the reference stroke800 of FIG. 8, the segmentation graph may include a path with thereference point 801 as an initial (first) node and a (second)consecutive node having the top-ranked (for example, the top five toten) combined key candidates based on the combined probability scoresfor all of the reference points 801-805. The combination of theprobability scores may be performed through application of certainfunctions, such as addition, multiplication, logarithmic, clustering,with or without weightings applied, such as in consideration of theafore-described characteristics of the input stroke 500 and/or referencestroke 800.

By this processing the input recognition system 114 provides at theoutput 120 the most likely text candidate(s), including words with orwithout grammatical marks, as described earlier. By this process, in theillustrated example the output 120 may include the candidate word of“everything” as the most likely recognition candidate based in theanalysis of the character selection stage 118.

While the foregoing has described what is considered to be the best modeand/or other examples, it is understood that various modifications maybe made therein and that the subject matter disclosed herein may beimplemented in various forms and examples, and that they may be appliedin numerous other applications, combinations, and environments, onlysome of which have been described herein. Those of ordinary skill inthat art will recognize that the disclosed aspects may be altered oramended without departing from the true spirit and scope of the subjectmatter. Therefore, the subject matter is not limited to the specificdetails, exhibits, and illustrated examples in this description. It isintended to protect any and all modifications and variations that fallwithin the true scope of the advantageous concepts disclosed herein.

We claim:
 1. A system for recognizing content associated withhandwritten stroke input to a computing device, the computing devicecomprising a processor, a memory and at least one non-transitorycomputer readable medium for recognizing content under control of theprocessor, the at least one non-transitory computer readable mediumconfigured to: detect input, with respect to an interactive key layoutdisplayed on an interface surface of a computing device, of ahandwritten stroke with respect to the interactive key layout;characterize the detected handwritten stroke by a reference stroke bydetermining a sequence of reference points associated with a sequence ofinteractive keys of the interactive key layout; assign one or moreprobability scores to one or more candidate characters associated witheach key of the interactive key sequence; and cause recognition of oneor more sequences of characters by applying a language model inaccordance with the assigned probability scores.
 2. A system accordingto claim 1, wherein each reference point is determined in relation tothe position of one or more points on the detected handwritten stroke toa characteristic of one or more of the interactive keys.
 3. A systemaccording to claim 2, wherein each reference point is a centroid of theone or more points.
 4. A system according to claim 1, the at least onenon-transitory computer readable medium configured to: determine one ormore combined probability scores of at least a subset of candidatecharacters of a combination of consecutive points of the sequence ofreference points; and cause recognition of one or more sequences ofcharacters by applying a language model in accordance with thedetermined combined probability scores.
 5. A method for recognizingcontent associated with handwritten stroke input to a computing device,the computing device comprising a processor, a memory and at least onenon-transitory computer readable medium for recognizing content undercontrol of the processor, the method comprising: detecting input, withrespect to an interactive key layout displayed on an interface surfaceof a computing device, of a handwritten stroke with respect to theinteractive key layout; characterizing the detected handwritten strokeby a reference stroke by determining a sequence of reference pointsassociated with a sequence of interactive keys of the interactive keylayout; assigning one or more probability scores to one or morecandidate characters associated with each key of the interactive keysequence; and causing recognition of one or more sequences of charactersby applying a language model in accordance with the assigned probabilityscores.
 6. A method according to claim 5, wherein each reference pointis determined in relation to the position of one or more points on thedetected handwritten stroke to a characteristic of one or more of theinteractive keys.
 7. A method according to claim 6, wherein eachreference point is a centroid of the one or more points.
 8. A methodaccording to claim 5, comprising: determining one or more combinedprobability scores of at least a subset of candidate characters of acombination of consecutive points of the sequence of reference points;and causing recognition of one or more sequences of characters byapplying a language model in accordance with the determined combinedprobability scores.
 9. A non-transitory computer readable medium havinga computer readable program code embodied therein, said computerreadable program code adapted to be executed to implement a method forrecognizing content associated with handwritten stroke input to acomputing device, the computing device comprising a processor, a memoryand at least one non-transitory computer readable medium for recognizingcontent under control of the processor, the method comprising: detectinginput, with respect to an interactive key layout displayed on aninterface surface of a computing device, of a handwritten stroke withrespect to the interactive key layout; characterizing the detectedhandwritten stroke by a reference stroke by determining a sequence ofreference points associated with a sequence of interactive keys of theinteractive key layout; assigning one or more probability scores to oneor more candidate characters associated with each key of the interactivekey sequence; and causing recognition of one or more sequences ofcharacters by applying a language model in accordance with the assignedprobability scores.
 10. A non-transitory computer readable mediumaccording to claim 9, wherein each reference point is determined inrelation to the position of one or more points on the detectedhandwritten stroke to a characteristic of one or more of the interactivekeys.
 11. A non-transitory computer readable medium according to claim10, wherein each reference point is a centroid of the one or morepoints.
 12. A non-transitory computer readable medium according to claim9, the method comprising: determining one or more combined probabilityscores of at least a subset of candidate characters of a combination ofconsecutive points of the sequence of reference points; and causingrecognition of one or more sequences of characters by applying alanguage model in accordance with the determined combined probabilityscores.